Lalonde (1986)

• National Supported Work experiment: qualified applicants randomly assigned to either a treatment or control condition. Treatment group received a guaranteed job for 9-18 months with additional counseling and support

Lalonde (1986)

Lalonde (1986)

• Lalonde swapped the control group for a sample from the general population, and demonstrated that a similarly constructed study with observational data would fail to find the correct result, even after controlling for observed characteristics

Lalonde (1986)

(he tried a bunch of stuff, none of it worked)

Lalonde (1986)

Candidates for job training were unlike the general population. They weren’t even especially similar to people with similar income, education, age, race, etc.

Observed Imbalance

Imbalance

Control variables probably can’t fix this.

• large observed differences often imply large unobserved differences
• even if all confounding is observed and controlled, regression controls assume a linear relationship between the confounders and the treatment effects.

Imbalance

Applied for program \(\rightarrow\) Earnings

Applied for program \(\leftarrow\) Education \(\rightarrow\) Earnings

Imbalance

dataFunction<-function(){
  N<-1000
  education<-rpois(N, 3) # education
  D<-rbinom(N, 1, arm::invlogit(education)) # applied to job program
  Y_1<- 2 + education^2 +  rnorm(100) # potential outcome for training
  Y_0<- 0 + education^2 +  rnorm(100) # potential outcome for no training
  ATE<-mean(Y_1-Y_0) # the true effect 
  Y<-ifelse(D==1, Y_1, Y_0) # the observed outcomes 
  df<-tibble("income" =Y, Y_0=Y_0, Y_1=Y_1,
             'treated' = factor(D, labels=c("Control", "Treated")), 
             education)
  return(df)
}

set.seed(9999)
data<-dataFunction()
ggplot(data, aes(x=income,y=treated)) + 
  stat_halfeye() +
  theme_bw() +
  xlab("Potential Outcome if Y(1)")

mean(data$Y_1 - data$Y_0)

[1] 1.870983

Imbalance

Notably, since the education confounder has a non-linear effect on income, we don’t actually get a correct estimate even if we control for it:

Matching

The experimental ideal allows us to assume this:

\[Y_i(0), Y_i(1), X_i \mathrel{\unicode{x2AEB}} D_i\] But we could theoretically get the average treatment effect if we can assume that the assignment is conditionally independent of expected outcomes.

\[Y_i(0), Y_i(1) \mathrel{\unicode{x2AEB}} D_i | X_i\]Both matching and regression methods attempt to meet this more relaxed condition, but the former can potentially do so more flexibly.

Exact matching

• Match groups with the exact same values, discarding levels of the predictor without at least one treated and one control unit.
• re-weight each strata so that the treatment and control groups are similar

Propensity score matching

Instead of matching on values, PSM matches observations with similar probabilities of receiving treatment.

• Estimate a model to predict being in the treatment group
• Group values with similar treatment probabilities
• Reweight the model so that the treated and control groups are similar
• Normally, discard observations outside of the “area of common support” where significant overlap exists.

Coarsened Exact Matching

Standard errors

Methods like CEM will generally group variables together before weighting, so we would expect to see clustering within the strata, but we know how to deal with this!

 Estimate Std. Error  z Pr(>|z|)     S 2.5 % 97.5 %
     1.88      0.157 12   <0.001 107.5  1.57   2.19

Term: treated
Type:  response 
Comparison: Treated - Control
Columns: term, contrast, estimate, std.error, statistic, p.value, s.value, conf.low, conf.high, predicted_lo, predicted_hi, predicted 

Takeaways

• Matching can address some of the limitations of regression models and simulate experimental conditions.

• Main advantages over regression are:

  • No assumption of a linear relationship between confounders
  • More informally: allows researchers to separate out the treatment selection process from the treatment effect process.